On-the-Fly Customization of Automated Essay Scoring

Because there is no commonly accepted view of what makes for good writing, automated essay scoring (AES) ideally should be able to accommodate different theoretical positions, certainly at the level of state standards but also perhaps among teachers at the classroom level. This paper presents a practical approach and an interactive computer program for judgment-based customization. This approach is based on the AES system, e-rater. Through this new approach, a user can gain easy accessibility to system components, flexibility in adjusting scoring parameters, and procedures for making scoring adjustments that can be based on only a few benchmark essays. The interactive prototype program that implements this approach allows the user to customize e-rater and watch the effect on benchmark essay scores as well as on score distributions for a reference testing program of the user’s choice. The paper presents results for the use of this approach in customizing e-rater to the standards of different assessments.